feat(rag): Option A escalation rescue + monitoring evidence gate parity#1040
Conversation
…59) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
…iss class) The monitoring-schedule answer gate was narrower than the fact filter and the eval judge: inflected schedules (reviewed annually, monitored for 3 hours), bare digit durations, and the level-range/metabolic-panel vocabulary passed fact filtering but died at the gate or at kind classification, so answers for those cases shipped without the asked-for schedule (run-#60 targeting misses). One shared monitoringScheduleEvidencePattern now feeds both the gate and the filter (lockstep by construction); the kind arm gains inflection tolerance only (panel/range vocab would steal sentences from the dose arm — pinned by the restored adjacent-context refusal test); and both the result-level and sentence-level coverage checks gain a figure escape mirroring the existing concrete-dose escape, tightly scoped to sentences/chunks that carry the asked-for interval or unit range themselves. Four red-proof tests (each verified failing pre-change) plus a negative guard pin the widen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
…up layer The neuroleptic-side-effect-escalation eval case failed the citation gate on every canary (#57-#60): the query misclassifies as medication_dose_risk, its honest lexical pool fails the dose fast-path floor (measured 0.184/0.0125 vs 0.66/0.055), the S3 document-lookup title layer that would surface the title-named SOP was excluded by the class allowlist, and the vector leg then injects the wrong sibling doc. shouldAttemptDocumentLookupFastPath now also engages the layer for medication_dose_risk when the classifier's existing deterministic signals say the query is escalation-shaped (intent escalation_risk — only assigned when drug_dosing wording did NOT match, so pure dose/route/frequency questions structurally cannot fire) AND a curated title alias phrase is present for the alias tier to rescue with (documentTitleTerms > 0). Rescue semantics by construction: the S3 block sits after both fast-path returns, so a medication_dose_risk query only reaches it once the floor already rejected the pool. Executed blast-radius sweep: exactly one eval case fires across the 44-case answer canary, the 36-case golden retrieval fixture, and the 30-case answer-quality harness — every non-firing case executes byte-identical code. No imputation formula, released-comparator key, selection clamp, or threshold is touched. Red-proof: the predicate test and the end-to-end rescue ordering test both fail with the gate reverted (verified); negative guards pin pure-dose, no-title, and non-escalation shapes plus the four allowlisted classes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe PR broadens monitoring-schedule evidence recognition across extractive-answer gates and adds an intent- and title-gated document lookup rescue for escalation-shaped medication-dose-risk queries, with regression tests and review ledger records. ChangesRAG evidence and retrieval
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Query
participant QueryAnalysis
participant RAGPipeline
participant DocumentLookup
participant ReleasedOrdering
Query->>RAGPipeline: submit escalation-shaped medication-dose-risk query
RAGPipeline->>QueryAnalysis: obtain intent and title terms
QueryAnalysis->>RAGPipeline: return escalation intent and curated title terms
RAGPipeline->>DocumentLookup: enable lookup for qualifying queries
DocumentLookup->>RAGPipeline: return S3 document candidates
RAGPipeline->>ReleasedOrdering: rank and stabilize candidates
ReleasedOrdering-->>Query: release rescued document ordering
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
… (review P2/P3)
rag-retrieval-reviewer reproduced a dose-intent regression from the parity
commit: the inflected monitoring kind tokens (monitor\w*, annually, LFTs)
claimed sentences carrying a concrete dose value ('Quetiapine is monitored at
a dose of 200 mg daily'), and dose-intent answers then rejected the
monitoring-kind fact — a source-gap where the figure previously shipped. The
kind arm now classifies the legacy tokens byte-identically and lets a sentence
claimed ONLY via the new inflections fall through to the dose arm when it
carries a clinicalDoseValuePattern value; both reviewer repro sentences are
pinned as dose-intent tests. Also extends the multi-drug bare-row guard to
monitoring_schedule (reviewer P3 symmetry): the figure coverage escape made
bare interval rows admissible without a query token, so a row naming no drug
inside a multi-drug chunk must not inherit the queried drug — red-proof
verified both directions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
…hain nit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/lib/rag/rag-extractive-answer.ts (1)
818-823: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winTighten the
monitoring_schedulegate fordosefacts
kind === "dose"is still allowed through this branch, and the current regex matches generic dose language liketherapeutic,range, andmaintenance. That lets sentences such as “The therapeutic dose range is 300-450 mg daily.” be extracted as monitoring-schedule evidence even though they contain no actual monitoring cadence/level signal. Require a real monitoring token fordose-kind facts before accepting them.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/rag/rag-extractive-answer.ts` around lines 818 - 823, Update the monitoring_schedule branch in the evidence-kind gate so dose facts are accepted only when the text contains an actual monitoring cadence or level token, rather than generic dose terms matched by monitoringScheduleEvidencePattern. Preserve the existing monitoring and renal_limit behavior, and reject dose text lacking that monitoring-specific signal.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/branch-review-ledger.md`:
- Line 676: Fix the MD037 spacing violation in the ledger row’s Markdown
emphasis markers, ensuring no spaces appear immediately inside emphasized text
delimiters. Preserve the row’s content and formatting otherwise, then run the
Markdown lint fixer or equivalent validation.
---
Outside diff comments:
In `@src/lib/rag/rag-extractive-answer.ts`:
- Around line 818-823: Update the monitoring_schedule branch in the
evidence-kind gate so dose facts are accepted only when the text contains an
actual monitoring cadence or level token, rather than generic dose terms matched
by monitoringScheduleEvidencePattern. Preserve the existing monitoring and
renal_limit behavior, and reject dose text lacking that monitoring-specific
signal.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 50a0f8d6-fd57-468d-ac4f-e81727d63b30
📒 Files selected for processing (5)
docs/branch-review-ledger.mdsrc/lib/rag/rag-extractive-answer.tssrc/lib/rag/rag.tstests/extractive-answer-formatting.test.tstests/rag-document-lookup-escalation-rescue.test.ts
…ignal CodeRabbit major on the parity widen: kind===dose facts passed the shared monitoring vocabulary via generic dose wording (range/therapeutic/maintenance around a mg figure), so 'The therapeutic dose range is 300-450 mg daily.' counted as monitoring-schedule evidence with no cadence/level signal. Dose-kind facts now additionally require a cadence or level token (monitoringCadenceOrLevelPattern — the strict schedule/level subset of the shared vocabulary); level ranges like 'Maintenance range is 0.6-0.8 mmol/L' stay admissible through their level units (red-proof pinned both directions). Also escapes the literal vector-* asterisks in the ledger row flagged MD037. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
|
CodeRabbit's outside-diff major (dose-kind facts passing the monitoring gate via generic dose vocabulary) is fixed in 57a898f: Generated by Claude Code |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/lib/rag/rag-extractive-answer.ts (1)
485-499: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winPrevent dose ranges from satisfying monitoring figure coverage.
monitoringFigureCoveragereuses a unit-range pattern that acceptsmg,micrograms, andmcg. Therefore, text such as “The therapeutic dose range is 300-450 mg daily” can satisfy the monitoring intent gate even though the dose-kind guard at Line 837 rejects it. Restrict monitoring figure coverage to concentration/level units or explicitly exclude concrete dose evidence, and add a regression test.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/rag/rag-extractive-answer.ts` around lines 485 - 499, Update monitoringFigureCoverage in the intent coverage logic to exclude concrete dose ranges accepted by monitoringUnitRangeFigurePattern, restricting matches to concentration/level units or applying the existing dose-kind guard before coverage is granted. Preserve valid monitoring schedule and interval evidence, and add a regression test confirming a therapeutic dose range such as “300-450 mg daily” cannot satisfy monitoring intent coverage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/lib/rag/rag-extractive-answer.ts`:
- Around line 485-499: Update monitoringFigureCoverage in the intent coverage
logic to exclude concrete dose ranges accepted by
monitoringUnitRangeFigurePattern, restricting matches to concentration/level
units or applying the existing dose-kind guard before coverage is granted.
Preserve valid monitoring schedule and interval evidence, and add a regression
test confirming a therapeutic dose range such as “300-450 mg daily” cannot
satisfy monitoring intent coverage.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: a014e33e-ef24-4a07-87dd-62e717d26003
📒 Files selected for processing (3)
docs/branch-review-ledger.mdsrc/lib/rag/rag-extractive-answer.tstests/extractive-answer-formatting.test.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/extractive-answer-formatting.test.ts
- docs/branch-review-ledger.md
…abbit major 2) The result-level and sentence-level monitoring coverage escapes reused monitoringUnitRangeFigurePattern, whose unit list includes dose-amount units (mg/micrograms/mcg) — so a therapeutic dose range could still grant monitoring intent coverage even though the dose-kind fact guard rejects the sentence itself. Both escapes now use monitoringLevelRangeCoveragePattern (mmol/L | nmol/L | mcg/L | ng/mL only); the wider pattern stays reserved for the corpus-guarded figure-promotion checks. Regression test pins a token-free chunk whose only candidate coverage grantor is a mg dose range. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
|
The second outside-diff major (dose-amount ranges satisfying the monitoring coverage escapes) is fixed in d6c218a: both the result-level and sentence-level escapes now use Generated by Claude Code |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/extractive-answer-formatting.test.ts`:
- Around line 575-584: Update the fixture in the test named “denies monitoring
intent coverage to a dose-amount range in a token-free chunk” so its content
contains only the dose range and neutral surrounding text, with no
monitoring-related or broader vocabulary such as “therapeutic,” “range,” or
“daily.” Preserve the assertion’s purpose of proving that “300-450 mg” alone
cannot grant monitoring coverage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: fdf35d51-b2d0-40b5-baa1-c5d6b8d416e1
📒 Files selected for processing (2)
src/lib/rag/rag-extractive-answer.tstests/extractive-answer-formatting.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/lib/rag/rag-extractive-answer.ts
…(CodeRabbit) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
Summary
The Option A wave (user-approved): kills the last blocking canary red and the answer-side half of the run-#60 monitoring targeting misses. Two single-concern changes plus a review-response hardening commit:
0abf3c9— Option A: title-supported escalation rescue (protected retrieval surface). Theneuroleptic-side-effect-escalationcase failed the citation gate on every canary (#57→#60): the query misclassifies asmedication_dose_risk, its honest lexical pool fails the dose fast-path floor (live-measured 0.184/0.0125 vs 0.66/0.055), the S3 document-lookup title layer was excluded by the class allowlist, and the vector leg injects the wrong sibling doc (Zuclopenthixol).shouldAttemptDocumentLookupFastPathnow also engages the layer formedication_dose_riskwhen BOTH existing deterministic classifier signals hold:intent === "escalation_risk"(assigned only when drug_dosing wording did NOT match — pure dose/route/frequency questions structurally cannot fire) anddocumentTitleTerms > 0(a curated title alias phrase is present for the alias tier to rescue with). Rescue semantics by construction: the S3 block sits after both fast-path returns, so a firing query has already had its pool floor-rejected. Blast radius proven twice over: exactly one eval case fires — my executed sweep and the retrieval reviewer's independent 110-case classifier probe agree. Honest live-expansion disclosure: real shapes like "When should clozapine be stopped?" now engage S3 (surfacing the title-named curated SOP for the named entity — assessed clinically correct; the principal thing the dedicated canary pair watches). No imputation formula, released-comparator key, selection clamp, or threshold is touched.1aebf02+a3b9a54— monitoring evidence gate parity (answer-side). Run-#60's four monitoring targeting misses traced to the answer-intent evidence gate being narrower than the fact filter and the eval judge: inflected schedules ("reviewed annually", "monitored for 3 hours"), bare digit durations, and level-range/metabolic-panel vocabulary passed fact filtering but died at the gate, at kind classification, or at the intent-token coverage checks. One sharedmonitoringScheduleEvidencePatternnow feeds both the gate and the filter (lockstep by construction), and both coverage checks gain a figure escape mirroring the concrete-dose escape, scoped to text carrying the asked-for interval or unit range itself. The review-response commita3b9a54closes both reviewer-found hazards: the inflected kind tokens no longer steal sole-dose-value sentences from the dose arm (legacy tokens classify byte-identically; new-inflection-only sentences fall through to dose when they carry a concrete dose value — both reviewer repro sentences pinned as tests), and the multi-drug bare-row guard extends tomonitoring_scheduleso a schedule row naming no drug inside a multi-drug chunk cannot inherit the queried drug (both governance-traced misattribution paths now refused, red-proven). Expected live effect:quality-lithium-monitoring-range+quality-antipsychotic-metabolic-monitoringgain their figures (monitoring targeting ≥3/5); the other two misses are retrieval-depth (documented residual).All four reviewer subagent verdicts (pre-ready, full ledger rows on branch):
a3b9a54, red-proven both directions.a3b9a54exactly as its smallest-fix prescribed; unsupported-figures/conservative-failure/adjacent-context/PHI all verified safe.55a6bc3).RAG impact: behaviour change — canary pair: run #60 (banked baseline on main 9b655fa) -> dedicated post-merge dispatch (answer_case_limit=44 + answer_quality_eval=true, ~$2-4, authorized envelope); success = citation_failure_rate 0.0227 -> 0, golden doc/content recall pinned 1.0/1.0, zero per-case rr@10 regressions, monitoring targeting ≥3/5, no other medication_dose_risk case's top-5 shifting; regression = single-commit revert of the offending commit per the proven drill.
Verification
npm run verify:cheap— all static checks green; unit suite 3087 passed / known container-only pdf artifactcheck:production-readiness— fails only on absent secrets in this demo container (expected per repo docs); hosted CI runs the CI-safe variantRisk and rollout
Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy) — no access changes🤖 Generated with Claude Code
https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
Summary by CodeRabbit
monitoring_scheduleevidence understanding with inflection-tolerant phrasing (cadence/follow-up/baseline) and direct interval/range figure coverage.medication_dose_riskwhen escalation intent and matching document title terms are present.monitoring_schedule.